Methods and systems for providing feedback and suggested programming methods

ABSTRACT

The techniques and supporting systems described herein provide a comprehensive and customizable approach to identifying the use of best practices during the design and development of software applications, as well as recommending additional enhancements or courses of action that may be implemented to further improve the application. Target software application code is received specific application security best practices applicable to the target software application are identified. Locations in the code where the various best practices ought to be implemented are then identified, and a determination is made whether the relevant best practices are implemented for each location. Finally, positive feedback is provided to the developers for what appears to be their correct implementation of best practices.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. provisionalpatent application Ser. No. 61/601,720, filed on Feb. 22, 2012, theentire disclosure of which is incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates generally to the field of computer programming,and more specifically to the assessment of programming techniques andadherence to programming standards for secure system design andapplication execution.

BACKGROUND

There are a myriad of testing and assessment techniques for validatingvarious properties of software applications and network implementations.However, one of the most critical processes for ensuring that thedeployment of software does not expose an organization to unacceptablerisks is security and vulnerability testing. Some of the conventionaltechniques used to perform such testing includes static analysis(automated code review), dynamic analysis (automated penetrationtesting) and manual analyses such as code review, design review, andmanual penetration testing. All of these analysis techniques are aimedat finding security weaknesses and vulnerabilities in an application andtypically provided in report format to the programmers, product managersand quality assurance (QA) staff. The report can provide detailedresults (e.g., program names, line numbers, variable names, dataconnections, etc.) as well as a summary of the results. The report maybe a conventional document such as a text file or a structured XML file.

To assist developers in steering clear of many of the well-knowpitfalls, system security professionals have developed, over time, anumber of best practices. These best practices are typically publishedas documents, text books, wiki pages or other reference materials. Thebest practices can include, for example, adherence to certain securecoding standards, use of enhanced-security code libraries, avoidance ofcode constructs or libraries known to be risky, etc.

There are a number of tools that attempt to identify potential or actualsecurity problems in application code, thus providing “negativefeedback” to the developers on suspect and, in some cases, suggestingpotential steps to improve the code. However, to date there have notexisted any automated mechanisms for explicitly identifying thedeveloper's affirmative use of more-secure best practices, or ofproviding “positive feedback” to the developer on their coding. As such,developers who implement certain well-designed coding or designtechniques may not fully benefit from a comprehensive knowledge baseregarding particular best practices.

The foregoing examples of the related art and limitations relatedtherewith are intended to be illustrative and not exclusive. Otherlimitations of the related art will become apparent upon a reading ofthe specification and a study of the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the sameparts throughout the different views. Also, the drawings are notnecessarily to scale, emphasis instead generally being placed uponillustrating the principles of the invention

FIG. 1 is a block diagram of a software security assessment platformaccording to an embodiment of the invention.

FIG. 2 is a flow chart depicting steps performed in software securityanalysis according to an embodiment of the invention.

DESCRIPTION OF THE INVENTION

The techniques and supporting systems described herein provide acomprehensive and customizable approach to identifying certain bestpractices used during the design and development of softwareapplications, as well as recommending additional enhancements or coursesof action that may be implemented to further improve the application. Asreferred to hereinafter, software applications may include (but are notnecessarily limited to) any sort of instructions for a machine,including, for example, without limitation, a component, a class, alibrary, an script, an applet, a logic table, a data block, or anycombination or collection of one or more of any one or more of these.

The appropriate type of software security analysis and best practiceimplementation depends on many factors, including (but not necessarilylimited to) the technical details of an application (e.g., the languagein which it is written and the platform on which is to be deployed) aswell as the business context in which the application operates. For anon-limiting example, an application that is “customer-facing” andfacilitates high-volume, secure transactions such as banking orecommerce will require rigorous testing to ensure that customer data isnot jeopardized. Conversely, applications such as document-controlsystems or desktop applications that are implemented entirely within anorganization and operated behind secure firewalls require less stringenttesting. Therefore, balancing the added costs for executing additionalsecurity assessments and testing with the risks of potential for lossesis critical.

FIG. 1 illustrates, in a broad overview, a representative softwaresecurity assessment platform 105 for implementing the techniquesdescribed herein. Although the diagrams depict components asfunctionally separate, such depiction is merely for illustrativepurposes. It will be apparent that the components portrayed in thisfigure can be arbitrarily combined or divided into separate software,firmware and/or hardware components. Furthermore, it will also beapparent that such components, regardless of how they are combined ordivided, can execute on the same host or multiple hosts, and wherein themultiple hosts can be connected by one or more networks.

In the example of FIG. 1, the platform 105 receives and reports onsoftware applications 110 from multiple entities, while monitoringnumerous sources 115 of external threats for up-to-date libraries ofmalware and application and environmental vulnerabilities. The platform105 includes a communications server 120 and an analysis engine 125. Asused herein, the term engine refers to software, firmware, hardware, orother component that is used to effectuate a purpose. The engine willtypically include software instructions that are stored in non-volatilememory (also referred to as secondary memory). When the softwareinstructions are executed, at least a subset of the softwareinstructions is loaded into memory (also referred to as primary memory)by a processor. The processor then executes the software instructions inmemory. The processor may be a shared processor, a dedicated processor,or a combination of shared or dedicated processors. A typical programwill include calls to hardware components (such as I/O devices), whichtypically requires the execution of drivers. The drivers may or may notbe considered part of the engine, but the distinction is not critical.

In the example of FIG. 1, the communications server 120 provides theconduit through which the platform interacts with external systems. Fora non-limiting example, the communications server 120 may utilizeconventional data-communications protocols such as TCP/IP, HTTP andothers to query application servers for updated application programs,download updated programs, post analysis results, and send and receivemessages from users. More specifically, in a server-basedimplementation, the communications server 120 may act as an interfacebetween the platform 105 and external entities that submit softwareapplications for assessment or review assessment results. In addition,the communications server 120 may act as a conduit through which otherexternal data such as updated threat information (in the form of malwaredefinition files, for example) are received for storage in the securitythreat database 150. In some implementations, the security assessmentplatform 105 may be configured as a distributed platform, in which oneor more components (e.g., testing modules, threat-assessment agents,secure communication devices, databases, etc.) are duplicated and/ordistributed among multiple computers located remotely from each otherbut, for example, co-located with users of the platform. Examples ofcommunications server application platforms providing such featuresinclude the Apache HTTP Web Server supplied by the Apache SoftwareFoundation and the WebSphere HTTP Server supplied by IBM Corporation.

In the example of FIG. 1, the analysis engine 125 receives applicationcode and programs of a target software application 110 from users,either via the communications server 120 or directly from customersusing the platform 105 as a subscription service. In otherimplementations, the application (or portions of the application) may bereceived from developers working within a development group, on adevelopment team, or as part of a community of otherwise unrelatedprogrammers. For example, the platform 105 may be implemented within alarge organization and used to scan software code as it is developed,checked in to a code repository, and/or put into production. In somecases, the platform 105 may be used as part of a check-in or postingprocess by a so-called “open soure” project, thus allowing the projectadministrator to suggest best practice coding techniques to projectcontributors.

In general, various embodiments of the analysis engine 125 provide asoftware system and methods of using the system that examine code of thetarget software application received and identify specific applicationsecurity best practices that are applicable to the target softwareapplication. The analysis engine 125 then identifies locations in thetarget application's code where the various best practices ought to beimplemented and determines for each location whether the relevant bestpractices appear to have been implemented. At each location, theanalysis engine 125 determines to what extent the relevant bestpractices appear to have been implemented correctly, and to what extentthey may be implemented incompletely or incorrectly and providespositive feedback to the developers for what appears to be their correctimplementation of best practices.

In some embodiments, the analysis engine 125 interacts with varioustesting engines and code review modules, as well with assessment andthreat databases, and includes benchmarking and reporting capabilitiesfor comparing assessment results among applications, developers, teamsand/or organizations.

In one embodiment, for a non-limiting example, the analysis engine 125interacts with a dynamic testing engine 130, a static testing engine135, a pen testing engine 140 and a module for performing manual codereview 145. In some embodiments, the dynamic analysis engine 130interacts with the target software application 110 as an external entityand executes the application 110 in a manner that mirrors or emulatesthe runtime environment in which it operates. In some embodiments, thedynamic analysis engine 130 receives a description of the interfaces tothe application 110, sends test and/or simulation data to theapplication via the interfaces, and analyzes the received responses. Thetest data may be application-specific (e.g., provided with theapplication as a library, data file, or structured input) orapplication-agnostic, such as data and/or scripts known to exploitapplication vulnerabilities. Based on the responses, the dynamicanalysis engine 130 determines whether any security defects exist in theapplication 110 and the extent to which it may be vulnerable to certainthreats. The defects and best practices may be reported in real-time(e.g., via the communications server 120) and/or stored in a databasefor subsequent analysis and reporting.

In some embodiments, the static analysis engine 135 receives a binary orbytecode version of the target software application 110 as input. Forexample, a high-level semantic model of the application 10 is createdcontaining control-flow and data-flow graphs of the application 110, andthis model then analyzed for the use of best practices and/or qualitydefects, including security flaws, by a set of analysis scans.

In some embodiments, the pen testing engine 140 performs penetrationtesting of the application 110. Penetration testing includes, forexample, simulating and analyzing various web-based interactions betweena client and the server on which the application 110 operates. Thisincludes executing standard HTTP commands such as GET and POST,analyzing FORM elements and scripting elements (both client andserver-side), and manipulating inputs to elicit known vulnerabilities.

In some embodiments, the analysis engine 125 may also receive input frommanual review processes executed using a manual code review module 145.Manual review processes typically include a human operator visuallyreviewing source code to determine if proper coding form and standardshave been followed, and looking for “extra” functions often left inapplications such as trap doors, Easter eggs, and similar undocumentedfunctionality.

In some embodiments, the data, scripts and functions used to identifythe best practices and operate the various testing engines and theanalysis engine 125 may be stored in a security-threat database 150. Thedatabase 150 may be operated as a stand-alone server or as part of thesame physical server on which the analysis engine 125 operates. Portionsof the threat database 150 may, in some cases, be provided by entitiesother than the entity operating the platform 105 on a subscriptionbasis, allowing the database 150 to be kept up to date as threats andmalware evolve over time. Likewise, the results of each test and theoverall analysis process may be stored in an assessment-results database155. In some embodiments, the applications and analysis results arestored in an encrypted format using a unique key provided to the ownerof the analyzed application 110 such that only it can access and reviewthe results of the analysis. In such cases, decryption of the analysisis limited to authorized personnel and all traces of the analysis aredeleted from memory (other than the database 155) following completion.Non-limiting examples of database applications that may provide thenecessary features and services include the MySQL Database Server by SunMicrosystems, the PostgreSQL Database Server by the PostgreSQL GlobalDevelopment Group of Berkeley, Calif., or the ORACLE Database Serveroffered by ORACLE Corp. of Redwood Shores, Calif.

In some embodiments, the examination of the target application by theanalysis engine 125 can be done through parsing of the applicationsource code, or the compiled bytecode (as in Java, .NET, and others) orbinary executable code (e.g. a compiled C/C++ application). In onenon-limiting instantiation, the analysis engine 125 examines theapplication through a combination of source code and binary codeparsing. In addition to parsing the structure of the program, theanalysis engine 125 constructs control-flow and data-flow graphs thatrepresent the behavior of the program. These “deep analyses” allow theanalysis engine 125 to analyze the intended and actual behavior of evencomplex, object-oriented programs.

In some embodiments, the analysis engine 125 may identify which bestpractices might or should apply to a particular target application in anumber of ways. In one particular embodiment, the mapping of the bestpractices that apply to an application may be expressed by the analysisengine 125 as a series of IF-THEN rules, such as “IF the targetapplication communicates with a database, THEN the database securityrules apply.” The rules may relate to the technical architecture orenvironment of the application (e.g., the operating system used, thecommunication protocol(s) used, whether encryption is used, etc.) and/orthe business implementation of the application. For a non-limitingexample, certain rules may be used to identify good coding practices forconsumer-facing financial transaction systems (ecommerce, banking, etc.)whereas others may be used for less risky back office applications suchas document management, workflow processing, inventory control, etc. Inanother embodiment, the use profile of an application (heavytransactional versus content delivery, mobile application etc.) maytrigger the application of certain rules. The rules may be applied bythe analysis engine 125 in a “forward-chaining” approach, a hierarchicalapproach, or a multi-path approach such that the applicability ofcertain rules may be dependant upon the evaluation of other higher-orderrules.

Once a set of rules has been identified that apply to a particularapplication, the analysis engine 125 identifies locations in the targetapplication where the certain best practices might be applied. Whilethere may be numerous methods for identifying the locations, theanalysis engine 125 may adopt one method, which uses a series ofpattern-matching rules, such as “At every location ‘L’ in the targetapplication where the target application sends a command string ‘S’ to adatabase, and where the command string ‘S’ could potentially includedata from an untrusted source ‘U’, a best practice dictates that aknown-good ‘cleanser function’ ‘F’ be applied to the command string ‘S’at a point in the dataflow between the untrusted source ‘U’ and thelocation ‘L’ where the command string ‘S’ is sent to the database.Furthermore, no additional untrusted data, whether from ‘U’ orotherwise, should be allowed to enter the command string ‘S’ betweenwhen it is cleansed and when it is sent to the database.” Theabove-described application can be modeled as:

Location ‘L1’:

Code §A. Untrusted data ‘U1’ enters the system

Code §B. . . . (cleanser function ‘F’ MUST be applied here) . . .

Code §C. A database command string ‘S’ is prepared, including data from‘U1’

Code §D. . . . (NO further untrusted data ‘U2’ may be allowed to enter‘S’)

Code §E. The command string ‘S’ is sent to the database

Based on the results of the evaluation of the rules, the analysis engine125 can determine whether the relevant best practices appear to havebeen implemented. Again, the approach depends on the particular bestpractice in question. Taking the example above, the analysis engine 125may scan the code of the target software application for the presence ofthe cleanser function ‘F’ in the sections of code ‘between’ (in adata-flow and control-flow sense, not merely in terms of physical codelayout) Code §A where the untrusted data ‘U1’ enters the system, andCode §C where the database command string ‘S’ is assembled. In thiscase, the presence of the cleanser function ‘F’ might indicate that thedeveloper had attempted to implement the relevant “best practice.”

To determine to what extent the relevant best practices appear to havebeen correctly and completely implemented may also depend on theparticular best practice in question. Taking the same example aboveagain, the analysis engine 125 scans the code of the target softwareapplication for common errors of correctness or completeness, such as(1) additional untrusted data ‘U2’ being added to the command string‘S’, or (2) control flow paths in the target application that causeexecution to jump from Code §A to Code §C, therefore bypassing executionof the cleanser function ‘F’.

In one embodiment of the methodology, if no implementation errors aredetected, the analysis engine 125 flags the implementation of the bestpractice using a moniker such as P+OK, “apparently present” (P) and“apparently correct” (OK). If implementation errors are detected, theimplementation of the best practice may be flagged as P+Err, “apparentlypresent” (P) and “incorrect” (Err), or as P+Inc, “apparently present”(P), and “incomplete” (Inc), or as P+Err+Inc, a combination of incorrectand incomplete implementation. If the wrong best practice for theparticular situation appears to have been implemented (e.g., cleanserfunction ‘F1’ is required, but cleanser function ‘F2’ is found instead),the implementation may be flagged as P+Mis, “apparently present”, and“mismatched for the situation” (Mis).

Once completed with the scan and analysis, the analysis engine 125provides positive feedback to developers for apparently complete andcorrect implementation of a “best practice.” Certain locations areflagged as P-OK (implementation apparently present and apparentlycorrect and complete) and communicated to the developers, either asindividual locations, or in aggregate, depending on the particular “bestpractice,” the number of instances occurring in the target application,and possibly other factors.

In some embodiments, the analysis engine 125 provides mixed positive andnegative feedback to the developers for locations where it appears thatthe developers attempted to implement a certain best practice, but theimplementation of the best practice is either incomplete or incorrect.Reports are provided the analysis engine 125 by identifying locationsflagged as P+Err, P+Inc, P+Err+Inc, or P+Mis to the developers, eitheras individual locations or in aggregate, depending on the particularbest practice, the number of instances occurring in the targetapplication, and possibly other factors. In effect, this informs thedeveloper that they have used correct security and coding practices butthe code needs additional work to implement these features in a mannerthat they have their full and intended effect.

Reporting to the developer may take many forms. As one non-limitingexample, the results may be reported in separate documents, emails, webpages, or other messages to the developer. In some cases, however, thereport may take the form of visual indicators added into the developer'sdevelopment environment such as green indicators or icons next to thelines of code where a best practice has been completely and accuratelyutilized, or shading the background of the “good” code light green.Other colors (yellow, red, etc.) may be used to indicate or highlightcode that used the best practices but needs additional attention, orplaces where no attempt was made to implement the best practices.

So as not to generate a surfeit of gratuitous positive feedback, andthus dilute the value of a small amount of well-deserved positivefeedback, the analysis engine 125 may explicitly excludes severalparticular situations. In particular, the analysis engine 125 may notgive positive feedback in situations where there is no need forimplementation of a best practice. If there is no actual security threatin a particular area of the code, there is no feedback given there,regardless of what the developer has implemented.

FIG. 2 is a flowchart illustrating a process 400 to implement andsupport software security analysis. Although this figure depictsfunctional steps in a particular order for purposes of illustration, theprocess is not limited to any particular order or arrangement of steps.One skilled in the relevant art will appreciate that the various stepsportrayed in this figure could be omitted, rearranged, combined and/oradapted in various ways.

In the example of FIG. 2, the flowchart 200 starts at block 202 wheretechnical characteristics of a target software application and businesscontext information relating to the target software application arereceived. The flowchart 200 continues to block 204 where code of thetarget software application received is examined and specificapplication security best practices that apply to the target softwareapplication are identified. The flowchart 200 continues to block 206where locations in the code of the target application where theidentified best practices ought to be implemented are identified andwhether the relevant best practices appear to have been implemented isdetermined for each of the locations. The flowchart 200 continues toblock 208 where whether the relevant best practices appear to have beenimplemented correctly and to what extent they have been implementedincompletely or incorrectly are determined at each of the locations. Theflowchart 200 ends at block 210 where positive feedback for what appearsto be their correct implementation of the best practices are provided todeveloper of the target software application.

Using various embodiments, the analysis engine 125 may “package” or“bound” the security analysis and vulnerability testing results to theactual software they describe. As used herein, “software” may refer to asingle program, a component, a page of mark-up language (e.g., HTML), aserver instance (either actual or virtual), a script, a collection ofprograms, or an entire application. In some cases, the software may be acommercially-available product delivered via traditional methods such asCD-ROM or download, whereas in other cases the software may be a websiteor collection of websites that provide the software and/or services overthe Internet, commonly referred to as software as a service, or “SaaS”.In still other cases, software may refer to a collective of otherwiseunrelated applications and services available over the internet, eachperforming separate functions for one or more enterprises, (i.e.,“cloud” computing). By linking the report to the software itself,downstream users of the software can access information about thesoftware, make informed decisions about implementation of the software,and analyze the security risk across an entire system by accessing all(or most) of the reports associated with the executables running on thesystem and summarizing the risks identified in the reports.

The methods and techniques describe above may be implemented in hardwareand/or software and realized as a system for producing, storing,retrieving and analyzing security and vulnerability reports for softwareapplications. For example, the platform 105 may be implemented as acollection of data processing modules for ingesting, reviewing, andattacking software applications, a module for producing the securityreport, and a data storage appliance (or series of appliances) to storethe reports. The platform 105 may in some cases also include a digitalrights management module that creates, certifies and confirms hashvalues, secure keys, and other user and use-based restrictive elementsthat ensure the authenticity of the users and that the reports are boundto the correct software components. In some embodiments the module mayset aside portions of a computer as random access memory to providecontrol logic that affects the processes described above. In such anembodiment, the program may be written in any one of a number ofhigh-level languages, such as FORTRAN, PASCAL, C, C++, C#, Java, Tcl, orBASIC. Further, the program can be written in a script, macro, orfunctionality embedded in commercially available software, such as EXCELor VISUAL BASIC. Additionally, the software could be implemented in anassembly language directed to a microprocessor resident on a computer.For example, the software can be implemented in Intel 80x86 assemblylanguage if it is configured to run on an IBM PC or PC clone. Thesoftware may be embedded on an article of manufacture including, but notlimited to, a computer-readable program means such as a floppy disk, ahard disk, an optical disk, a magnetic tape, a PROM, an EPROM, orCD-ROM, appliances, partitions (either physical or virtual) of singleappliances, or combinations of the two.

One skilled in the art will realize the invention may be embodied inother specific forms without departing from the spirit or essentialcharacteristics thereof. The foregoing embodiments are therefore to beconsidered in all respects illustrative rather than limiting of theinvention described herein.

What is claimed is:
 1. A software security assessment platform,comprising: a communications server, which in operation, receivestechnical characteristics of a target software application and businesscontext information relating to the target software application; ananalysis engine, which in operation: examines code of the targetsoftware application received and identifies specific applicationsecurity best practices that are applicable to the target softwareapplication; identifies locations in the code of the target applicationwhere the identified best practices ought to be implemented anddetermines for each of the locations whether the relevant best practicesappear to have been implemented; determines at each of the locationswhether the relevant best practices appear to have been implementedcorrectly and to what extent they have been implemented incompletely orincorrectly; and provides positive feedback to developer of the targetsoftware application for what appears to be their correct implementationof the best practices.
 2. The platform of claim 1 further comprising adynamic analysis engine that executes the target software application ina manner that mirrors or emulates the runtime environment in which itoperates.
 3. The platform of claim 1 further comprising a staticanalysis engine that receives a binary or bytecode version of the targetsoftware application as input and creates a high-level semantic model ofthe application containing control-flow and data-flow graphs of theapplication for analysis.
 4. The platform of claim 1 further comprisinga pen testing engine that performs penetration testing of the targetsoftware application.
 5. The platform of claim 1 further comprising amanual code review module that receives input from manual reviewprocesses including a human operator visually reviewing the code of thetarget software application to determine if proper coding form andstandards have been followed, and looking for “extra” functions oftenleft in the application.
 6. The platform of claim 1 further comprising abenchmarking and reporting module that compares assessment results amongapplications, developers, teams and/or organizations.
 7. The platform ofclaim 1 further comprising a digital rights management engine thatapplies a digital rights management process against applicationassessment input data, thereby limiting distribution and use of thevulnerability test results to specified users.
 8. The platform of claim1 wherein the analysis engine provides mixed positive and negativefeedback to the developers for locations where it appears that thedevelopers attempted to implement a certain best practice, but theimplementation is incomplete or incorrect.
 9. The platform of claim 1wherein the analysis engine examines the target software applicationthrough parsing of one or more of the source code, the compiledbytecode, and binary executable code of the target software application.10. The platform of claim 1 wherein the analysis engine maps the bestpractices that are applicable to the target software application as aseries of IF-THEN rules, wherein the rules are applied in a“forward-chaining” approach, a hierarchical approach, or a multi-pathapproach such that the applicability of certain rules is dependent uponthe evaluation of other higher-order rules.
 11. The platform of claim 1wherein the analysis engine identifies locations in the code of thetarget application where the identified best practices ought to beimplemented by using a series of pattern-matching rules.
 12. Theplatform of claim 1 wherein the analysis engine determines at each ofthe locations whether the relevant best practices appear to have beenimplemented by scanning the code of the target software application forthe presence of cleanser functions.
 13. The platform of claim 1 whereinthe analysis engine determines at each of the locations to what extentthe relevant best practices have been implemented incompletely orincorrectly by scanning the code of the target software application forcommon errors of correctness or completeness.
 14. The platform of claim1 wherein the analysis engine provides positive feedback to developersof the target software application by flagging the implementation of thebest practices based on whether or not implementation errors aredetected.
 15. The platform of claim 1 wherein the analysis engineprovides mixed positive and negative feedback to the developers forlocations where it appears that the developers attempted to implement acertain best practice, but the implementation of the best practice iseither incomplete or incorrect.
 16. The platform of claim 1 wherein theanalysis engine does not provide positive feedback in situations wherethere is no need for implementation of a “best practice.”
 17. Theplatform of claim 1 wherein the analysis engine does not provide anyfeedback if there is no actual security threat in a particular area ofthe code regardless of what the developer has implemented.
 18. A methodfor software security assessment, comprising: receiving technicalcharacteristics of a target software application and business contextinformation relating to the target software application; examining codeof the target software application received and identifying specificapplication security best practices that are applicable to the targetsoftware application; identifying locations in the code of the targetapplication where the identified best practices ought to be implementedand determining for each of the locations whether the relevant bestpractices appear to have been implemented; determining at each of thelocations whether the relevant best practices appear to have beenimplemented correctly and to what extent they have been implementedincompletely or incorrectly; and providing positive feedback todeveloper of the target software application for what appears to betheir correct implementation of the best practices.
 19. The method ofclaim 18 further comprising executing the target software application ina manner that mirrors or emulates the runtime environment in which itoperates.
 20. The method of claim 18 further comprising receiving abinary or bytecode version of the target software application as inputand creating a high-level semantic model of the application containingcontrol-flow and data-flow graphs of the application for analysis. 21.The method of claim 18 further comprising performing penetration testingof the target software application.
 22. The method of claim 18 furthercomprising receiving input from manual review processes including ahuman operator visually reviewing the code of the target softwareapplication to determine if proper coding form and standards have beenfollowed, and looking for “extra” functions often left in theapplication.
 23. The method of claim 18 further comprising comparingassessment results among applications, developers, teams and/ororganizations.
 24. The method of claim 18 further comprising applying adigital rights management process against application assessment inputdata, thereby limiting distribution and use of the vulnerability testresults to specified users.
 25. The method of claim 18 furthercomprising providing mixed positive and negative feedback to thedevelopers for locations where it appears that the developers attemptedto implement a certain best practice, but the implementation isincomplete or incorrect.
 26. The method of claim 18 further comprisingexamining the target software application through parsing of one or moreof the source code, the compiled bytecode, and binary executable code ofthe target software application.
 27. The method of claim 18 furthercomprising mapping the best practices that are applicable to the targetsoftware application as a series of IF-THEN rules, wherein the rules areapplied in a “forward-chaining” approach, a hierarchical approach, or amulti-path approach such that the applicability of certain rules isdependent upon the evaluation of other higher-order rules.
 28. Themethod of claim 18 further comprising identifying locations in the codeof the target application where the identified best practices ought tobe implemented by using a series of pattern-matching rules.
 29. Themethod of claim 18 further comprising determining at each of thelocations whether the relevant best practices appear to have beenimplemented by scanning the code of the target software application forthe presence of cleanser functions.
 30. The method of claim 18 furthercomprising determining at each of the locations to what extent therelevant best practices have been implemented incompletely orincorrectly by scanning the code of the target software application forcommon errors of correctness or completeness.
 31. The method of claim 18further comprising providing positive feedback to developers of thetarget software application by flagging the implementation of the bestpractices based on whether or not implementation errors are detected.32. The method of claim 18 further comprising providing mixed positiveand negative feedback to the developers for locations where it appearsthat the developers attempted to implement a certain best practice, butthe implementation of the best practice is either incomplete orincorrect.
 33. The method of claim 18 further comprising not providingpositive feedback in situations where there is no need forimplementation of a “best practice.”
 34. The method of claim 18 furthercomprising not providing any feedback if there is no actual securitythreat in a particular area of the code regardless of what the developerhas implemented.